Binocular Disparity Review and the Perception of Depth

quantitative mapping of binocular receptive fields of the We perceive the world in three-dimensions even though cat visual cortical cells and suggested models for simuthe input to our visual system, the images projected lating their responses. While these and many other exonto our two retinas, has only two spatial dimensions. periments have demonstrated the neural substrates for How is this accomplished? It is well known that the disparity coding at the earliest stage of binocular convisual system can infer the third dimension, depth, from vergence, they leave open the question of how a populaa variety of visual cues in the retinal images. One such tion of disparity selective cells could be used to compute cue is binocular disparity, the positional difference bedisparity maps from a pair of retinal images such as the tween the two retinal projections of a given point in stereograms used by Julez. What is needed, in addition space (Figure 1). This positional difference results from to experimental investigations, is a computational thethe fact that the two eyes are laterally separated and ory (Marr, 1982; Churchland and Sejnowski, 1992) specitherefore see the world from two slightly different vanfying an algorithm for combining the neuronal signals tage points. into a meaningful computational scheme. The idea that retinal disparity contributes critically Although there have also been many computational to depth perception derives from the invention of the studies of stereo vision in the past, until recently, most stereoscope by Wheatstone in the 19th century, with studies had treated disparity computation mainly as a which he showed conclusively that the brain uses horimathematics or engineering problem while giving only zontal disparity toestimate the relative depths of objects secondary considerations to existingphysiological data. in the world with respect to the fixation point, a process Part of this tradition stems from a belief advanced paraknown as stereoscopic depth perception or stereopsis. doxically by David Marr, one of the most original thinkers Because simple geometry provides relative depth given in vision research, that physiological details are not imretinal disparity, the problem of understanding stereo portant for understanding information processing tasks vision reduces to the question: How does the brain mea(such as visual perception) at the systems level. Marr sure disparity from the two retinal images in the first (1982) argued that a real understanding will only come place? from an abstract computational analysis of how a particSince Wheatstone’s discovery, students of vision sciular problem may be solved under certain mathematical ence have used psychophysical, physiological, and assumptions, regardless of the neuronal implementacomputational methods to unravel the brain’s mechations in the brain. Although the importance of Marr’s nisms of disparity computation. In 1960, Julez made computational concept cannot be overstated, the main the important contribution that stereo vision does not problem with ignoring physiology is that there is usually require monocular depth cues such as shading and permore than one way to “solve” a given perceptual task. spective (see Julez, 1971). This was demonstrated Without paying close attention to physiology, one often through his invention of random dot stereograms. A comes up with algorithms that work in some sense but typical stereogram consists of two images of randomly have little to do with the mechanisms used by the brain. distributed dots that are identical except that a central In fact, most previous stereo vision algorithms contain square region of one image is shifted horizontally by a nonphysiological procedures that could not be implesmall distance with respect to the other image (see Figmented with real neurons (see Qian, 1994, for a disure 6a for an example). When each image is viewed cussion). individually, it appears as nothing more than a flat field To understand visual information processing perof random dots. However, when the two images are formed by the brain instead of by an arbitrary machine, viewed dichoptically (i.e., the left and right images are one obviously has to construct computational theories presented to the left and right eyes, respectively, at the of vision based on real neurophysiological data. Such same time), the shifted central square region “jumps” a realistic modeling approach to stereo vision has been out vividly at a different depth. This finding demonstrates proven possible recently. Disparity-tuned units, based that the brain can compute binocular disparity without on the response properties of real binocular cells, can much help from other visual modalities. be shown to effectively compute disparity maps from The first direct evidence of disparity coding in the stereograms. Moreover, the stereo algorithm can be exbrain was obtained in the late 1960s, when Pettigrew tended to include motion detection and provide coherand coworkers recorded disparity selective cells from ent explanations for some interesting depth illusions the striate cortex in the cat, the primary visual area